Bayes Optimal Hyperplanes! Maximal Margin Hyperplanes

Maximal margin classifiers are a core technology in modern machine learning. They have strong theoretical justifications and have shown empirical successes. We provide an alternative justification for maximal margin hyperplane classifiers by relating them to Bayes optimal classifiers that use Parzen windows estimations with Gaussian kernels. For any value of the smoothing parameter (the width of the Gaussian kernels), the Bayes optimal classifier defines a density over the space of instances. We define the Bayes optimal hyperplane to be the hyperplane decision boundary that gives lowest probability of classification error relative to this density. We show that, for linearly separable data, as we reduce the smoothing parameter to zero, the Bayes optimal hyperplane converges to the maximal margin hyperplane. We also analyze the behavior of the Bayes optimal hyperplane for linearly non-separable data, showing that it satisfies a very natural optima. We explore the idea of using the hyperplane that is optimal relative to a density with some small non-zero kernel width and compare the resulting hyperplane with the maximal margin and the soft margin hyperplanes.